Multitenant hosted virtual machine infrastructure

ABSTRACT

A multi-tenant virtual machine infrastructure (MTVMI) allows multiple tenants to independently access and use a plurality of virtual computing resources via the Internet. Within the MTVMI, different tenants may define unique configurations of virtual computing resources and unique rules to govern the use of the virtual computing resources. The MTVMI may be configured to provide valuable services for tenants and users associated with the tenants.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is related to and incorporates by reference in theirentirety U.S. Non-Provisional Application entitled MULTITENANT HOSTEDVIRTUAL MACHINE INFRASTRUCTURE (U.S. patent application Ser. No.12/434,619) filed May 2, 2009 and U.S. Non-Provisional Applicationentitled MULTITENANT HOSTED VIRTUAL MACHINE INFRASTRUCTURE (U.S. patentapplication Ser. No. 12/434,621) filed May 2, 2009. This application isa divisional of U.S. Non-Provisional application Ser. No. 12/434,620,filed May 2, 2009 and claims the benefit of U.S. Provisional PatentApplication No. 61/050,163, filed on May 2, 2008, and U.S. ProvisionalPatent Application No. 61/101,665, filed on Sep. 30, 2008, each of whichis hereby incorporated by reference in its entirety.

BACKGROUND

A virtual machine is a software construct designed to run computerprograms like a real physical machine. As an example, a virtual machinemay comprise various software components for running executable codewritten for a particular computing platform. Such executable code mayinclude, among other things, operating system code, applicationprograms, software drivers, and so on.

A virtual machine does not necessarily have a direct correspondence to areal physical machine; however, virtual machines are often designed toimitate the behavior of specific hardware architectures. As an example,a virtual machine may be designed to imitate the behavior of an x86processor.

Virtual machine infrastructures (VMIs) have been developed to coordinateuser access to collections of virtual machines and to provide additionalfeatures such as virtual networking capability, virtual data storagemanagement, and so forth. In general, a VMI comprises a set of physicalcomputational resources, virtualization software for creating virtualmachines on the physical computational resources, and managementsoftware for managing the virtual machines and coordinating access tothe virtual machines.

In one example, a VMI comprises a set of physical computers each runningmultiple virtual machines. Users at separate remote-access consolesaccess the different virtual machines over a local area network. Eachconsole includes a virtual machine interface designed to allow a user tointeract with a virtual machine in the same way that the user wouldinteract with a local machine. For instance, the virtual machineinterface may present a user desktop and explorer windows similar tothose found in an ordinary personal computer.

Within a VMI, different virtual machines may be individually configuredaccording to the users' unique needs and preferences. For instance, thedifferent virtual machines may run different types of operating systems(e.g., Windows, Linux), allowing the users to use differentoperating-system-specific programs within the VMI. Additionally, each ofthe virtual machines may provide a different level of performance sothat the resources of the single physical computer can be efficientlydivided among users having different computational demands.

Recently, large enterprises have begun employing complex VMIs to providevirtual computing resources for large groups of users. These users maywork together, but have different computational demands. As an example,a company with hundreds of employees may set up a virtual data centercomprising many physical machines each configured to run several virtualmachines for use by the employees.

The virtual machines can be configured in accordance with the differentcomputational demands and preferences of the different employees. Forinstance, an employee whose job requires a significant amount ofcomputing power—say, an engineer who runs test simulations on complexcircuits—may use a virtual machine configured with higher throughput andmore memory, compared with an employee whose job only requires the useof a few simple programs such as a word processor and an Internetbrowser. Similarly, an employee whose job requires a relatively highlevel of system access—say, a system administrator—may use a virtualmachine configured with a higher level of access within the VMI comparedwith an employee whose job requires a relatively lower level of access.

Conventional VMIs are designed for use within a single organization,i.e., a single company or group of users. In this type of VMI, theoperation of the virtual machines is governed by a common set of rules,such as a common hierarchy of access rights, usage rights, quality ofservice guarantees, and naming conventions. Additionally, in this typeof VMI, the administration of the computing hardware and operatingsoftware as well as the configuration and execution of virtual machineson this infrastructure are controlled by a single administrative entity,i.e., a single system administrator or group of system administrators.Furthermore, these conventional VMIs do not provide adequately isolatedor independent network services or storage services for virtual machinesexecuted by different users. Also, these conventional VMIs do notsupport adequate resource usage quota, reporting, and enforcementmechanisms.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a conceptual diagram providing a system-level overview of amultitenant virtual machine infrastructure (MTVMI) in some embodiments.

FIG. 2 is a flowchart illustrating the use of a MTVMI by a new tenant insome embodiments.

FIG. 3A is a flowchart illustrating the use of a MTVMI by a userassociated with a particular tenant in some embodiments.

FIG. 3B is a display diagram illustrating an example browser used to loginto a MTVMI and a resulting screen that appears in the user's displayupon logging in to the MTVMI in some embodiments.

FIG. 4 is a flowchart illustrating the operation of a MTVMI in relationto a tenant in some embodiments.

FIG. 5 is a flowchart illustrating the operation of a MTVMI in relationto a user in some embodiments.

FIG. 6 is a conceptual diagram providing a more detailed system-leveloverview of a hosted multitenant virtual machine infrastructure in someembodiments.

FIG. 6A is a block diagram illustrating a storage manager in someembodiments.

FIG. 7 is a flowchart illustrating the provision of a software brokeringservice by an MTVMI in some embodiments.

FIG. 8 is a flowchart illustrating the provision of a virtual labservice by an MTVMI in some embodiments.

FIGS. 9-25 are display diagrams showing a typical user interfacepresented by an example software application running on a MTVMI toprovide a virtual lab service in some embodiments.

FIGS. 25-26 show additional conceptual diagrams illustrating theorganization of the MTVMI in some embodiments.

FIGS. 27-46 are display diagrams showing displays presented by the MTVMIin some embodiments.

FIGS. 47-49 are display diagrams illustrating a user interface designedfor use by vendors offering items for sale via the brokering portal insome embodiments.

FIG. 50 is a block diagram illustrating a data model representation ofvirtual data centers in the MTVMI management application in someembodiments.

FIG. 51 is a block diagram illustrating component processes in the MTVMImanagement application in some embodiments.

FIG. 52 is a block diagram illustrating the organization of the datamodel that supports access control mechanisms of MTVMI managementapplication in some embodiments.

FIG. 53 is a block diagram illustrating the organization of the datamodel that supports resource usage quota mechanisms of MTVMI managementapplication in some embodiments.

FIG. 54 is a flowchart illustrating the operations performed by acomponent of the MTVMI management application to instantiate a virtualdata center configuration on the MTVMI infrastructure in someembodiments.

FIG. 55 is a flowchart illustrating the operations performed by acomponent of the MTVMI management application to save the state of theVirtual Data Center in some embodiments.

DETAILED DESCRIPTION

A variety of embodiments will now be described. These embodiments areprovided as teaching examples and should not be interpreted to limit thescope of the invention. Although specific details of the embodiments arepresented, these embodiments may be modified by changing, supplementing,or eliminating many of these details.

A shared computing infrastructure (“the infrastructure”) is described.In some embodiments, a user can exploit the entire infrastructure usinga standard web browser running on a personal client machine (e.g., theuser's laptop), without having to install any specialized applicationson the client machine. The infrastructure supports the deployment ofvirtual data centers, wherein each virtual data center comprises one ormore virtual machines capable of transparently interfacing amongstthemselves and with computing resources located elsewhere in theInternet. As an example, the infrastructure may execute a virtual datacenters on physical machines within the infrastructure, while userscontrol the virtual machines through personal computers connected to theinfrastructure via the Internet.

The shared computing infrastructure may incorporate a softwareenvironment and interface capable of facilitating the automation ofspecific virtualized infrastructure based services. As an example, theinfrastructure may incorporate a software platform that implements aportal where users can develop and exchange data that may be usefulwithin the infrastructure. This data could include, e.g., virtual datacenter configurations and software applications that could be run withinthe virtual data center. The data could be developed by individual usersor by group collaboration within the infrastructure. Additionally, thedata could be exchanged with or without monetary payments.

For explanation purposes, the shared computing infrastructure will bereferred to as a multi-tenant virtual machine infrastructure (MTVMI). AMTVMI is a VMI adapted for use by multiple tenants, where each “tenant”is an independent entity (e.g., a set of users within a single companyor other organization) whose access to and use of the MTVMI is governedby a unique and independent set of rules. For instance, each tenant in aMTVMI may have a unique and independent authentication mechanism, accesscontrol model, hierarchy of rights and privileges, resource allocationcontrol model, resource usage control, quotas, service level agreements,and billing. The MTVMI may provide virtual data center isolation foreach tenant. For example, the MTVMI may allocate a number of virtualresources for the exclusive use of a tenant concerned with security oravailability of services. While tenants have access to and use of theMTVMI, the MTVMI may be owned and operated by a different independententity (logically the “landlord”).

Although the term “tenant” generally has a connotation related to therental or occupation of property or space, “tenant” in the currentcontext should not be limited to entities having certain legal orlogistical agreements or arrangements relative to the MTVMI; neithershould it be limited to entities who occupy a certain quantity of spaceor resources within a MTVMI. Rather, “tenant” here merely refers to anyentity having a certain logical relationship with the MTVMI asdemonstrated by the examples and explanations presented in this writtendescription.

Examples of multiple tenants sharing a MTVMI include different companiesor groups of individual users. For instance, an MTVMI could be shared bymultiple large corporations each having its own set of virtual datacenters, application software assets, set of users and so on.

The MTVMI is unique in delivering virtual data centers as a service overthe Web based on a multi-tenant shared infrastructure. In addition tothose mentioned above, the MTVMI may provide additional capabilitiesvalued by customers in the initial target markets (e.g., QualityAssurance, Staging, Training, Pre-sales Proof of Concept, etc.), such asnetwork services (e.g., DHCP, DNS) for virtual data centers,connectivity to internal data centers, rapid creation of custom virtualmachines, storage virtualization, and remote access to virtual datacenters and virtual machine environments.

FIG. 1 is a conceptual diagram providing a system-level overview of anMTVMI in some embodiments. MTVMI 100 comprises a physical infrastructureresident at the MTVMI provider's site, virtual infrastructure 105, asoftware library 110, and a MTVMI management application 115.

The left side of FIG. 1 shows different tenants 130 and 140 connected toMTVMI 100 through the Internet 120. Different users among tenants 130and 140 may access MTVMI 100 over corresponding secure Internetconnections. Typically, a user accesses MTVMI 100 by logging on to theinfrastructure through a console or Internet browser. Once the user logson to MTVMI 100, a user interface for MTVMI 100 appears in the user'scomputer display to allow the user to access various features of MTVMI100. In some embodiments, the user interface appears similar to a normalcomputer desktop, so that the user's experience is substantially thesame as it would be if the user was interacting with a set of localresources. The user interface enables a user associated with a tenant tomanage virtual data centers, such as by creating, configuring, starting,and stopping virtual machines within the virtual data centers. In someembodiments, the user interface may control a user's access to certainresources of the virtual data center. For example, the user interfacemay allow users with administrative privileges to start and stop virtualmachines while prevent other users from doing so.

In general, tenants can access MTVMI 100 over the Internet from anyInternet-enabled device. As examples, the MTVMI can be accessed throughpersonal computers, Internet-enabled personal digital assistants (PDAs),and Internet-enabled cell phones. In some embodiments, where a useraccesses the MTVMI 100 using a portable device such as a PDA or cellphone, the user does so with a web browser installed in the device, muchlike the user would do in a personal computer.

Each tenant of MTVMI 100 operates within its own unique, private virtualenvironment. The environment for each tenant may include, for example,one or more private virtual data centers each comprising a set ofvirtual machines, virtual storage, and virtual network accessible to thetenant's users. The environment for each tenant may provide the tenant'susers with access to ready-to-instantiate templates of virtual datacenters and specific software applications in Software Library 110.Additionally, the environment may provide Labs corresponding toinstantiated virtual data centers that can be executed. Furthermore,through the MTVMI, a tenant may create custom virtual data centers byspecifying certain attributes of the virtual data centers and/or theirvirtual machines (e.g., number of cores, amount of RAM, domain name, IPaddresses, etc.). The MTVMI provides a number of methods for buildingand rapidly deploying virtual machines, such as uploading and installingcomplete virtual machine images, installing operating system packages onraw virtual machine containers, installing application packages onpre-built virtual machine images, etc. Furthermore, because the MTVMImay store the state of an entire virtual data center, including itsvirtual machines, storage, network identifies, network service profiles,and so on, the virtual data centers and their virtual machines can bereplicated. The environment may further include methods for managing thevirtual data centers, such as a hierarchy of user access rights, limitson the amount of resources such as central processing unit (CPU) timeand storage space available to the users, and so on. In someembodiments, the MTVMI can provide resource isolation for some or all ofits tenants. For example, the MTVMI may assign a tenant with heightenedsecurity concerns a set of virtual resources for the exclusive use ofthat tenant. When the tenant is processing or manipulating secure data,the tenant can use the “private” resources to prevent other tenants ofthe MTVMI from comprising the tenant's secure data.

Within MTVMI 100, virtual infrastructure 105 comprises both a physicalplatform for running virtual machines, virtual storage, and virtualnetwork, and a software platform enabling the creation and management ofthe virtual machines, virtual storage, and virtual network. The physicalplatform may comprise computer hardware such as storage system andmedia, servers comprising CPUs, memory, and related processingcomponents, and networking hardware such as network switches, routers,etc. The software platform may include, e.g., a hypervisor for eachserver, and software components for the storage system and networkingcomponents and software for managing them.

Within virtual infrastructure 105, different functional components canbe roughly partitioned into three groups, including network nodes,virtual machine host nodes, and storage nodes. Network nodes includeboth hardware and software components for implementing virtual networkfunctionality (e.g., network services such as DHCP and DNS, VPNservices, file share services, etc.), virtual machine nodes includehardware and software components for creating and hosting virtualmachines, and storage nodes include hardware and software components forimplementing virtual data storage. In some embodiments, the storagenodes are part of a distributed file system management architecture thatsupports, among other things, fast snapshot and cloning operations in ascalable way, such as data placement, data migration, and loadbalancing.

Software library 110 comprises a collection of software programs, dataobjects, and/or virtual data center configurations for use by tenantsand associated users who interact with MTVMI 100. The software programsmay include, among other things, specific operating systems to be run onvirtual machines, application programs to run within the virtualmachines, and any code used by tenants to maintain their own virtualresources. The data objects may include, for example, database files,software configuration files, and so forth. The virtual data centerconfigurations may include, e.g., specifications of network topologies,and hardware and software specifications for individual virtual machineswithin virtual networks. One example configuration for a virtual machinecould include 2 GHz virtual CPU and 1 GB RAM, and run Windows XP.

Some of the programs and data in software library 110 may be commercialsoftware requiring individual or corporate user-licenses. The licensingof these programs or data may be regulated in any of several differentways. As examples, tenants or individual users could provide their ownlicense keys for the programs, or licenses could be provided as part ofa tenant's agreement for using the MTVMI. In some embodiments, MTVMI 100includes mechanisms for automatically negotiating the licenses withthird-party software providers in real time. In some embodiments,licenses are purchased by the entity hosting the MTVMI 100 so that usersare not required to obtain their own licenses. In some embodiments,software library 110 may be connected to a licensing module to negotiateon-demand licensing of proprietary software to tenants, or management ofa limited number of existing licenses among multiple users.

In some embodiments, software library 110 is divided into public andprivate libraries. A public library is a portion of software library 110accessible to any tenant using MTVMI 100. A private library, on theother hand, is a portion of software library 110 accessible to only asubset of the tenants using MTVMI 100 or to a subset of users belongingto a tenant. Public libraries may include, for instance, freewaresoftware applications, commercial operating systems and applicationsthat can be provided to tenants for a charge based on usage, shareddata, and so on. Private libraries may include, for instance, licensedor proprietary software for use by individual tenants, specific dataobjects related to work performed by the tenant or the configuration ofa tenant's virtual network. In some embodiments, a library can contain acommercial operating system, application, or other program that, wheninstalled on a virtual machine, will be customized with a license keyowned by the installing tenant. In some embodiments, the MTVMI makescommercial library items available to users on a pay-per-use basis. Inthese embodiments, the MTVMI provides the architecture, data model,workflows, and metering to support this service. The MTVMI enablesproducers of such content to upload and identify their products so thatconsumers can find them and download them for use. Furthermore, theMTVMI can meter, enforce, and bill consumers for any period of usage.

In addition to specific software programs and data structures, softwarelibrary 110 may further include specific virtual data centerconfigurations for use by individual tenants. A virtual data centerconfiguration stored in the software library contains completeinformation for instantiating a set or “cluster” of virtual machinesconnected by a virtual network. The virtual data center configurationspecifies capabilities of each virtual machine to be instantiated,including such attributes as number of processor cores, amount ofmemory, operating systems, etc. The virtual data center configurationfurther contains complete state for each virtual machine to beinstantiated. Such state generally includes contents of a disk volume tobe accessible by the virtual machine, and may also include memorycontents of the virtual machine, and/or processor state for the virtualmachine, including contents for registers including program counters. Insome embodiments, a user may upload a virtual machine image generatedoutside the MTVMI. This virtual machine image can be stored in thelibrary, and instantiated by the MTVMI in a virtual data center.

An instance of a virtual data center can be created by a tenant's userusing the content in the software library. As an example, one virtualdata center may include several virtual machines with a particularoperating system and particular applications, as well as virtualstorage, that is together performing a batch application. A tenant or atenant's user may have a number of such virtual data centerconfiguration instances, which can be in varying states, includingrunning and stopped. The set of these virtual data center configurationinstances comprise the Lab for the tenant or the tenant's user.

Further descriptions of software libraries and labs are provided belowwith reference to FIGS. 11-20. Various additional functions and benefitsof software library 110 will be readily apparent from the descriptionsof FIGS. 11-20.

MTVMI management application 115 controls and monitors interactionsbetween tenants and MTVMI 100. Additionally, MTVMI managementapplication 115 allocates resources among different tenants andcoordinates the use of MTVMI 100 by the different tenants. Variousmanagement related tasks performed by MTVMI management application 115may include, as examples, authenticating users, allocating CPU time andstorage space among different tenants, maintaining logical isolationbetween different tenants, tracking each tenant's usage of MTVMI 100,and many others. Additional examples of specific functions that can beperformed by MTVMI management application 115 are described below inrelation to FIG. 6.

Tenant information such as virtual data center configurations and storeddata within MTVMI 100 may be generated in a variety of different ways.For instance, in some embodiments, the tenant information may begenerated by a tenant's privileged user through manual operations on theMTVMI 100 using its graphical interface. Alternatively, virtual datacenter configurations and other tenant information can be generated forMTVMI 100 by providing a virtual machine appliance container thatdescribes an existing physical or virtual machine to a module withinMTVMI 100, and operating the module to generate a virtual data centerconfiguration for use within MTVMI 100. Moreover, virtual data storagefacilities can be established and initialized within MTVMI 100 bydownloading data from an existing data storage platform into MTVMI 100.

FIG. 2 is a flowchart illustrating the use of a MTVMI by a new tenant insome embodiments. FIG. 2 is intended to provide a very broadillustration of the tenant's interactions with the MTVMI, beginning withthe initial establishment of a usage agreement between the tenant and anMTVMI host. In the description that follows, it can be assumed thatactions performed by the tenant are actually carried out by arepresentative of the tenant, such as a system administrator or anotherauthorized individual.

Referring to FIG. 2, in step 205, a tenant establishes a relationshipwith a MTVMI host by submitting a subscription request to the host. Inthe subscription request, the tenant may specify, among other things, adesired amount of resource usage within the MTVMI, any desired featuressuch as access to particular software applications, and other terms forgoverning tenant's use of the MTVMI. A tenant's resource usage can bemeasured in any of several different ways, including, e.g., by a totalamount of CPU time used by the tenant, or by a total amount of time thatusers are logged on to the MTVMI.

Once a tenant has subscribed to use the MTVMI, the tenant may populatetheir MTVMI account in step 210. They may create individual accounts forthe tenant's users. They may also populate the account with data such asvirtual machine images, software installation binaries, data files etc.by transferring them as needed from the tenant's other computingenvironments or elsewhere.

As one of the steps in establishing practices for using their MTVMIaccount, in step 215, the tenant establishes rules to govern use of theMTVMI environment. Examples of these rules include, among others,creating project structures to organize the environment, assigning usersdifferent privileges on different projects to establish access control,and establishing limits on the amount of resource usage such ascomputational bandwidth and storage space allocated to different users.The access rights, in addition to limiting use of certain resources bycertain users, may also limit the ability of certain users to changeaspects of the virtual data center, e.g., by adding additional machines,and so forth. In many regards, the rules governing the virtual datacenter can be established so they are similar to the rules that governordinary, physical data centers. The MTVMI provides the environment andtools for a tenant's users to establish these governance rules.

Once rules have been established for the virtual data center, in step220, the tenant's users can establish virtual data centers within theMTVMI. Each of the tenant's virtual data centers may include, amongother things, a set of virtual machines, a set of virtual networks, anda set of data storage resources. The performance and capacity of thetenant's virtual data centers may be limited by the tenant'ssubscription agreement; however, the tenant is generally free toconfigure the logical structure of the data center in any way desired.At a high level, the tenant's process of establishing a virtual datacenter may be compared, by analogy, to the process that an organizationgoes through to establish a physical data center, by purchasing a set ofphysical resources, connecting the resources together, and getting theresources to communicate with one another. However, in contrast to aphysical data center, a virtual data center requires significantly lesslegwork on the part of the system architects, and therefore tends torequire much less time and expense.

Once resources have been configured for the virtual data center, in step225, the tenant may continue to perform maintenance and monitoring onits virtual data center while the virtual data center is being used bythe tenant's users. The maintenance and monitoring may include upgradingand patching various components, modifying the set of users associatedwith the tenant, measuring the resource usage by different users, and soon.

Those skilled in the art will appreciate that the steps shown in FIG. 2and in each of the flow charts discussed below may be altered in avariety of ways. For example, the order of the steps may be rearranged;some steps may be performed in parallel; shown steps may be omitted, orother steps may be included; etc.

FIG. 3A is a flowchart illustrating the use of a MTVMI by a userassociated with a particular tenant in some embodiments. To use theMTVMI, in step 305, the user opens an Internet browser. As explainedabove, the Internet browser may be included in any type of web-enableddevice, including, e.g., a personal computer, a web-enabled cell phone,or a PDA.

After opening the Internet browser, the user enters a URL for the MTVMIto load an MTVMI access portal website into the browser. In step 310,the user logs onto the MTVMI through the access portal by supplyingcredentials such as a user name and password to the portal.

After the user logs onto the MTVMI, in step 315, the MTVMI beginsexecuting resources on behalf of the user so that the user may access,for example, the software library, labs, and data store. A lab can havemultiple virtual data center configuration instances, each of which canhave multiple virtual machines. The display of the user's machine may bemodified to show a desktop for any instantiated virtual machine in thelabs. The desktop may resemble an ordinary computer desktop as isnormally displayed on a personal computer. As a result, the user mayinteract with the virtual machine in essentially the same way that theuser would interact with a local machine.

FIG. 3B is a display diagram illustrating an example browser used to loginto a MTVMI and a resulting screen that appears in the user's displayupon logging in to the MTVMI in some embodiments. Through the browserwindow, the user can navigate to the console display 355 of one of theirvirtual machines. As seen in FIG. 3B, within this virtual machine, theuser may have access to a virtual local disk 365, and documents 360stored in a virtual storage platform. In addition to this virtualmachine, the user may also have access to other virtual machines withina virtual data center created by the tenant. In one embodiment, theseadditional virtual machines may be accessible through an icon such asthe “My Network Places” icon 370 displayed on the screen 355.

Once the user is logged onto the MTVMI, the user may interact with thevirtual machine in much the same way that the user would interact with anon-virtual local machine. In addition, as will be described in greaterdetail with reference to FIGS. 7-22, a user who has logged onto theMTVMI may also perform actions such as creating a lab with multiplevirtual machines, and installing custom software applications on thedifferent virtual machines for specific tasks.

FIG. 4 is a flowchart illustrating a method of operating a MTVMI inrelation to a tenant in some embodiments. The method of FIG. 4 can beviewed as a counterpart to the method illustrated in FIG. 2. In otherwords, while FIG. 2 illustrates actions performed by a tenant inrelation to the MTVMI, FIG. 4 shows actions performed by the MTVMI inrelation to a tenant. In general, the actions of the MTVMI may beperformed by a MTVMI management application such as that illustrated inFIGS. 1 and 6.

Referring to FIG. 4, in step 405, the MTVMI receives a subscriptionrequest from a new tenant. As described above in relation to FIG. 2, thesubscription request may include, among other things, informationspecifying desired usage rights for the new tenant. In step 410, theMTVMI generates account data for the new tenant based on the informationprovided with the subscription request. The account data may include,e.g., information for regulating MTVMI access and usage by the tenant,such as a data storage quota, authorized users associated with thetenant, a CPU quota, network bandwidth quota, and user authenticationinformation.

After the account data is generated, in step 415, the MTVMI allocatestenant resources, such as data storage space, CPU time, and so forth. Insome embodiments, these resources are allocated on an on-demand basis.In other words, resources such as storage space and CPU time are onlyallotted to a tenant when the tenant performs an operation requiring theresources. In other embodiments, however, resources may be reserved foruse by individual tenants, regardless of whether the tenants currentlyneed resources.

FIG. 5 is a flowchart illustrating the operation of a MTVMI in relationto a user in some embodiments. The method of FIG. 5 can be viewed as acounterpart to the method illustrated in FIG. 3. In other words, whileFIG. 3 illustrates actions performed by a user in relation to the MTVMI,FIG. 5 shows actions performed by the MTVMI in relation to a user.

Referring to FIG. 5, in step 505, the MTVMI receives user login and/orauthentication information. Based on this information, the MTVMIdetermines whether to allow the user to access the MTVMI. Upon grantingthe user access to the MTVMI, in step 510, the MTVMI allows the user toconfigure a virtual data center. Upon request from the user toinstantiate a virtual data center configuration, in step 515, the MTVMIbinds the selected virtual machines to one or more virtual machine hostsby communicating with the hyperviser for the virtual machine hosts. Instep 520, the MTVMI allows the user to perform authentication of anylicenses for selected software. In step 525, the MTVMI begins executingthe resources. During the execution of virtual machines within MTVMI,the MTVMI may run a console application to generate information on theuser's display. In particular, the user can view and interact with thedesktop console of the virtual machine in a display such as that shownin FIG. 3B.

FIG. 6 is a conceptual diagram providing a more detailed overview ofMTVMI 100 shown in FIG. 1 in some embodiments. Further to thedescription of FIG. 1, FIG. 6, shows various example components that maybe included in virtual infrastructure 105, software library 110, labs112, and MTVMI management application 115.

In the example of FIG. 6, virtual infrastructure 105 comprises aplurality of network nodes 605, a plurality of virtual machine hostnodes 610, and a plurality of storage nodes 615, each of which can beimplemented using either general purpose commodity server hardware orserver hardware more specialized to the roles of network processing,virtual machine execution, and high-volume reliable storage,respectively.

Although the description treats these different nodes as separatefeatures, it should be recognized that the functionality of these nodescan be implemented by different hardware and software arrangements.Moreover, the functionality of the different nodes will not necessarilybe implemented within different physical or logical structures. In otherwords, some components may implement functions for more than one type ofnode.

Each of the network nodes 605 comprises one or more physical or virtualmachines that establish one or more virtual LANs for each virtualnetwork specified by one of the labs 112. A virtual network connectstogether the virtual machines of a network, and enables them to accesstheir corresponding virtual data stores on the storage nodes. Inparticular, to implement a virtual LAN, the network nodes establish anindependent set of network services for each virtual LAN. These networkservices include such network services as virtual private network,firewall, network address translation, dynamic DNS, dynamic hostconfiguration protocol service, and traffic shaping. In someembodiments, one or more of these services can be disabled for aparticular virtual network. In some embodiments, the user may configurea virtual data center to specify whether, for each of these networkservices, a standard version is executed in the network node, or aspecialized version is executed in the virtual machine. In a similarmanner, the user can configure other networking aspects of the virtualLAN, such as by setting static IP addresses for the virtual machineswhich are private only to the virtual LAN or which are publicly routablefrom the public Internet, connecting individual VPNs from each virtualmachine to their own corporate data center, collectively connecting allthe virtual machines in a configuration through a VPN gateway to theirown corporate data center, setting their own domain name, running theirown domain name server, etc. In some embodiments, one or more of thesenetwork nodes work cooperatively as a load-balancing cluster or afailover cluster.

The organization and configuration of network nodes and the networkingattributes of individual virtual machines and appropriate virtualstorage devices guarantee that network services for each virtual datacenter are isolated. Network interfaces of virtual machines areconfigured to operate within one or more specific virtual LANs. Networkservice instances (e.g., name resolution, DHCP service, VPN gatewayservice, network file share service etc.) are operated independently foreach of these virtual LANs and each of these instances are restricted toservice network traffic only from specific virtual LANs. In addition,network nodes are configured with appropriate rules to enforce therouting of network traffic to appropriate VLANs, to mask private networkaddresses through network address translation, and to firewall or openspecific network ports and map between specific ports. The MTV/MImanagement application makes these configuration changes on the networknodes and the individual virtual machines in an automated way based onthe input received from the user on the desired networking attributesfor the virtual data center.

Each of virtual machine host nodes 610 comprises one or more physicalmachines capable of executing one or more virtual machines. Eachphysical machine has a corresponding hypervisor and the one or morevirtual machines runs on top of the hypervisor.

Each of storage nodes 615 comprises one or more physical data storagemedia. These storage media may provide virtual storage for virtualmachines running within the MTVMI. Additionally, storage nodes mayprovide long and/or short term data storage for tenant information suchas virtual data center configurations, and data archives. In variousalternative embodiments, storage nodes 615 may be populated by users ortenants directly interacting with MTVMI 100, or through automaticprocesses such as automatically downloading content stored in anotherdata storage system. Additionally, in some embodiments these storagenodes may support capabilities to create snapshots, clones, and mirrorsfor data objects stored on these nodes. The snapshot of a data objectcreates a point-in-time checkpoint copy of the data object that is notdirectly modifiable. The cloning of a data object creates apoint-in-time copy of the data object that can subsequently modifiedindependent of the parent object. The mirror of a data object creates aduplicate copy of the data object that is maintained as an exact replicaof the parent object even if the parent object is modified subsequently.In some embodiments the copies created by the snapshot, cloning, ormirroring of a data object are created on the same physical storage nodewhere the parent data object resides. In other embodiments, these copiesmay be created on a different storage node. Additionally, in someembodiments the copies created by snapshot and cloning create aphysically distinct copy. In other embodiments, these copies aremaintained as logical copies of the parent data object and actualphysical copies are created on demand when content is modified either inthe parent or the copy (e.g., using a well-known Copy-on-Writemechanism). The MTVMI infrastructure can take advantage of these variousembodiments to realize powerful unique capabilities. For example, whenthe state of all the virtual machines in a virtual data center (e.g.,this includes the state of the virtual machines' processors, memory, anddisk) is maintained on the storage nodes, the MTVMI may provide afeature to create an identical but independent copy of the virtualmachines in a virtual data center rapidly by executing operations on thestorage node to create clones of the data objects that maintain thevirtual machine states on the storage node. In combination with othersteps, e.g., to create identical but independent virtual networks, theMTVMI may support the capability to rapidly clone a virtual data centerconfiguration.

Management application 115 comprises components for managing andmonitoring user access to MTVMI 100. As shown in FIG. 6, for instance,management application 115 may comprise an account management componentfor managing a tenant's account data, a remote console component fordisplaying a graphical user interface in a user's local display, amonitoring and metering component for tracking resource usage bydifferent tenants and users, a backup component for maintaining backupcopies of the data stored by storage nodes 615, an audit logs componentfor maintaining audit logs documenting all the actions performed by eachtenant's users, a network manager component for controlling the creationand functioning of isolated virtual networks and network services withinthe MTVMI 100, a VM host manager component for controlling the creationand functioning of virtual machines within MTVMI 100, and a storagemanager component for controlling the creation and functioning ofvirtual storage within the MTVMI 100 and for coordinating tenant anduser access to virtual storage services.

In some embodiments, the storage manager component of the MTVMImanagement application is responsible for storage-related operations(e.g., backup, relocation, replication). In some embodiments, thestorage manager centralizes storage knowledge within the MTVMImanagement application, provides unified management of basic storageoperations spanning multiple or dissimilar storage products, andinsulates the rest of the application from the numerous details of theactual storage implementation. The storage manager supports portabilityacross a wide range of lower-level storage implementations and enablesan administrator to deploy different Common off the Shelf (COTS) storageproducts on the MTVMI management application. The storage manager mayoptimize use of the underlying storage nodes by, for example, delegatingcertain storage operations to certain storage nodes based on thecapacity/usage, load, and health of the storage nodes. The storagemanager may also isolate a specific tenant's use of storage to aspecific subset of storage nodes to meet performance, security, oravailability considerations of the tenant. The storage manager alsosupports improved supportability and maintainability, redundant datastorage for improved availability, and horizontal scalability.

In some embodiments, the storage manager uses load-balancing techniquesto select for each virtual machine the storage node that is assigned toprovide virtual storage to the virtual machine. In some embodiments, thestorage manager further provides intelligent data placement, dynamicdata replication, and/or dynamic data migration. Intelligent dataplacement involves strategically selecting one or more storage nodes inwhich to store particular data. For example, an image for a popularoperating system may be placed on multiple storage nodes in order toprovide for high expected demand for the operating system among virtualmachines. Dynamic data replication involves creating additional copiesof particular data on different nodes in order to provide betterperformance for accessing the data. Dynamic data migration involvesmoving data to a different storage node, such as to relieve abottleneck. Typically, these mechanisms are implemented by monitoringand collecting data about the behavior of host nodes and/or storagenodes and associated workload levels, such as which objects are moreused than others, and policy intelligence for making determinationsbased upon this input. In some embodiments, the storage nodes providevirtual network attached storage devices for use by virtual machinesrunning within the MTVMI.

In some embodiments, the storage manager uses copy-on-write techniquesthat represent a child image stored at a later time for particularvirtual machine as a set of changes made to a parent image stored forthat virtual machine at an earlier time. In such embodiments, adependency exists in the child image on the parent image. Where theparent image is to be moved to a different storage node, the storagemanager may migrate the child image with the parent image, or it maybreak the dependency of the child image on the parent image bytransforming the representation of the child image into a complete copyof the child image.

FIG. 6A is a block diagram illustrating a storage manager in someembodiments. Storage manager 650 is comprised of client processes 655embedded with client libraries (not shown), agent processes 660 and 665,and a state database 680. The storage manager defines a distinct layeron top of the underlying storage nodes 670 and 675 and presents aunified view across these storage nodes.

Client processes 655 execute storage management intelligence which,among other things, determines how to allocate responsibilities toindividual storage nodes. For example, the client processes determine:for particular data that is to be stored, on what storage node(s) tostore it; when to move particular data from one storage node to another;when to replicate data on one storage node to another; and, in responseto a retrieval request for particular data that is stored on each ofmultiple storage nodes, which node to retrieve the data from. In makingsuch determinations, the client processes consider content,capacity/usage, performance, load, health, and affinity information thatthe storage manager collects and maintains for each of the storagenodes. In many cases, determinations made by client processes involvedetermining a subset of feasible nodes, then selecting a node from amongthe feasible nodes.

If new content must be stored that has a dependency on pre-existingcontent (e.g., creating a copy or snapshot of existing content), thenthe storage nodes that have that pre-existing content are the subset offeasible nodes. If content must be retrieved, then the subset of nodesthat have that specific content is the feasible subset. In selecting onenode from among the set of feasible nodes, the client processes seek toavoid nodes that are less healthy (e.g., their disks are in a degradedstate) and to prefer nodes that have lower usage and load. Other choicesare possible. In particular, tenants of the service can be assigned anaffinity to a specific subset of nodes (and vice versa where a node isassigned affinity to a subset of tenants), such that requests on behalfof that tenant will target this subset of affinity nodes.

Logic to determine which content should be replicated or migrated and towhich node is also based on the same attributes as above. The logicadditionally schedules these operations during a window when there isless performance impact. In addition, system administrators can manuallytrigger replication/migration of specific data to specific nodes.

Client libraries embedded within client processes provide local-APIaccess to the storage nodes, manage the overall state of the storagesubsystem, act as a communication channel between the client processesand the agents, and collect performance and load data for the agents.

Agent processes 660 and 665 run on or near the storage nodes and act asan adapter that supports a common API and semantics across the differentunderlying storage nodes. The agent processes provide a layer toabstract the details of the specific storage nodes from the storagemanager, which provides the MTVMI management application with a unifiedinterface for interacting with any number and/or type of storage nodes.In this example, agent processes 660 provide an interface to storagenodes 670 of type “Storage Node Type 1” while agent processes 665provide an interface to storage nodes 675 of type “Storage Node Type n.”

State database 680 stores state of the storage subsystem and its variouscomponents and allows concurrent operations to proceed safely. Forexample, state database 680 may store state information for each filestored in storage nodes 670 and 675. When a client process performs anoperation on a file, such as a write operation, the state database may“lock” the file so that other client processes do not performinconsistent operations on the file simultaneously. In this manner,storage manager 650 may use state database 680 to serialize conflictingoperations.

In some embodiments, management application 115 further includes abilling system component for monitoring a tenant's use of resourceswithin MTVMI 100 and producing an invoice. The monitoring may occur, forexample, by tracking the amount of CPU time or data storage consumed bythe users associated with each tenant.

In some embodiments, management application 115 further comprises aresource integration module for facilitating inclusion of external(i.e., non-local or other third party) resources into MTVMI 100. In thisway, MTVMI 100 can be used to provide potentially brokered access tocomputing, storage and network resources regardless of where theresources are physically located. Such access can be provided accordingto policies for achieving any number of tenant or user objectives, suchas, e.g., cost, reliability, security, availability, and performance.

In some embodiments, management application 115 further includes amodule for specifying a grid configuration of a virtual data centerbased on input from a tenant. A grid configuration is a specification ofthe topology of a set of virtual machines and related networks andstorage systems. The grid configuration module could generate the gridconfiguration by receiving a tenant's description of an existingphysical or virtual network in the form of a tenant-created file or datastructure, and then modifying the description into a form usable byMTVMI 100. The grid configuration can subsequently be imaged onto MTVMI100 for use by particular tenant or user.

Management application 115 may further comprise an access component forallowing tenants to restrict the rights of users to specific physical orvirtual resources, configurations, and other software resources withinMTVMI 100. For instance, when establishing an account for MTVMI 100, atenant could specify a set of users with access to a particular softwaresuite within MTVMI 100, and a set of users without access to thesoftware suite. The MTVMI management application additionally couldinclude a security component that could monitor access to a softwaresuite based on the tenant's specification. Additionally, the securitycomponent could issue alerts or perform other actions in response toattempts at unauthorized access, etc.

In some embodiments, management application 115 further comprises aquota component allowing tenants to modify resource quotas amongindividual users associated with the tenant, or among all usersassociated with the tenant. More generally, the policy component couldallow tenants to modify any aspect of the tenant's subscriptionagreement with MTVMI 100.

In some embodiments, management application 115 further comprisesmaintenance components such as patching and virus protection softwarefor updating different the configurations of tenants' different virtualnetworks, backup or replication software for storing redundant copies ofvirtual machines, application packages, installation and testing toolsand services, etc. In general, these maintenance components may operatewith or without input from tenants or users, with or without oversightby the entity hosting MTVMI 100, and in a fully automated,partially-automated, or non-automated manner.

In components allowing tenant or user interactions with MTVMI 100, suchinteractions may take place through a variety of different interfaces,such as graphical interfaces, command line interfaces, programminginterfaces, and so on.

Management application 115 may further include custom monitoring andmanagement components created by tenants or users. To implement thesecustom features, a user could create scripts or application programsusing a scripting program or an application programming interface (API)compatible with MTVMI 100. To facilitate the creation of such programs,management application 115 may additionally include a softwaredevelopment platform for MTVMI 100. By allowing tenants and users tocreate custom software components, MTVMI 100 may provide flexibility andunique capabilities useful to a variety of users having a variety ofneeds.

In addition to the above-described components and features, managementapplication 115 may further comprise software applications for providingspecific virtualization-based services to tenants and users within MTVMI100. Examples of such services will be described below with reference toFIGS. 7 and 8. A more specific example of the service of FIG. 8 willthen be described with reference to FIGS. 9-22.

Several data structures and computing structures used to implementcapabilities of the MTVMI management application, in some embodiments,are described in further detail below with reference to FIGS. 50-53.

In various embodiments, the MTVMI and the portal together support avariety of uses, including application development and testing; onlineeducation; pre-sales software demonstration; consulting platform; and IPprototyping and operations.

FIG. 7, which illustrates the provision of a brokering service by MTVMI100 in some embodiments, is described in further detail below.

FIG. 8 is a flowchart illustrating the provision of a virtual labservice by an MTVMI in some embodiments. In the example of FIG. 8, theuser creates a virtual lab and executes a software test using thevirtual lab. A virtual lab is a set of virtual computing resources thatcan be configured and re-configured frequently in the process ofprototyping, demonstrating, validating, or offering training on one ormore computing tasks. As an example, a virtual lab may comprise a set ofvirtual machines and related software configured to stream video to anumber of diverse client machines. This virtual lab may be particularlyuseful for validating that the streamed media is rendered accurately onclient machines that differ in their operating system or applicationsoftware. Other example steps that could be performed within a virtuallab application will be apparent from the description presented below inrelation to FIGS. 9-22.

Referring to FIG. 8, in step 805, a user accesses a virtual labapplication within MTVMI 100. The virtual lab application may generate agraphical user interface on the user's display, such as the interfaceillustrated in FIGS. 9-22. Upon accessing the virtual lab application,in step 810, the user creates or accesses one or more projects. Aproject is a data organization folder that is assigned a set ofresources or objects in MTVMI (e.g., virtual data center configurations)and a set of users that can access the resources assigned to theproject. In some embodiments, resource usage is separately measured andquota compliance is separately enforced for each project. By specifyingboth users and resources, the project allows specific virtual datacenter instances to be developed through a collaborative process amongthe users. Example projects will be described in further detail belowwith reference to FIGS. 9-10. In some embodiments, a tenant'sadministrative user can specify resource permissions at a variety ofdifferent levels of scope, including private scope in which permissionsare established for a single user, or tenant-wide scope in whichpermissions are established for all users for a particular tenant.Resource permissions can also be specified at all-tenant scope, in whichpermissions are established for users of all tenants. Such permissionsmay relate to items from the software library, the labs, or asset listsestablished by or for tenants.

After establishing the project, in step 815, the user creates a lab byselecting and configuring particular resources within the project. As anexample, the user may select a set of virtual machines from the projectto create a virtual data center or virtual lab and install softwareapplications to be run on the virtual machines. Example labs will bedescribed in further detail below with reference to FIGS. 9 and 11-12.Multiple virtual data centers, identical or otherwise, can be created bydifferent users.

Once the project and lab have been created, in step 820, the user runsthe lab to perform a particular computational task—in this case, asoftware test. An example of a software test performed in a virtual labwill be presented below with reference to FIGS. 9-22. Multiple users canconcurrently perform the software test on a virtual data center thatthey all have access to or they can perform the software testconcurrently on independent virtual data centers. At the end of thetests, users can snapshot the entire state of the virtual data center,if needed, to save the outcome of the tests on the virtual data center.In some embodiments, this snapshot copy of the virtual data center canbe assigned to the same project, allowing other users in the project toinstantiate and examine the virtual data center to analyze the resultsof the testing.

FIGS. 9-22 are display diagrams showing a user interface presented by anexample software application running on a MTVMI to provide a virtual labservice in some embodiments. The virtual lab service is availableon-demand over the Internet and allows users to rapidly build andconfigure software test environments using a pre-built library ofvirtual machine images. The virtual lab service further allows users tocreate virtual infrastructure such as virtual processing, networking,and storage facilities on demand using a browser. Finally, the virtuallab service allows users to share data and collaborate with other userswho are also connected to the virtual lab environment via the Internet.

The operation of the virtual lab service will be explained below using aconcrete example involving an engineer who uses a MTVMI to test ane-commerce website on behalf of a tenant having users located indifferent parts of the world. The engineer may share components of thevirtual lab with the other users associated with the tenant.

FIG. 9 illustrates a screen that appears after the engineer logs on,through a secure connection, to a MTVMI hosting the virtual lab service.In this regard, the screen of FIG. 9 corresponds to an entry point forthe virtual lab service. As seen in FIG. 9, the screen includes a tablabeled as dashboard 905.

Dashboard 905 includes information related to current projects andrecently used items within the engineer's scope of user privileges. Inparticular, dashboard 905 includes a set of projects 910, virtual datacenter configurations in software library 915, and virtual data centerconfiguration instances in lab 920 associated with the engineer. Theprojects 910, software library 915, and lab 920 of FIG. 9 may be createdwithin the illustrated application, or they may have been createdpreviously.

In the current example, we will assume that the engineer will deploy avirtual data center configuration instance that was previously created.To do this, the engineer may simply select the virtual data centerconfiguration from the list 920 shown under the heading “My Lab”. As anexample, to select the lab illustrated in FIG. 11, the engineer mayselect a virtual data center configuration 921 entitled “Company A—Build1340”. Similarly, to select the virtual data center configurationillustrated in FIG. 12, the engineer may select a virtual data centerconfiguration 922 entitled “Company A—Build 1341”. By selecting one ofthese configurations, the user can bring up the screen shown in FIG. 11or FIG. 12, which will be described in further detail below.

In testing the e-commerce site, the engineer verifies the site'sscalability and performance on virtual machines defined in the selectedlab. A realistic test of a e-commerce site typically requires a largenumber of machines. However, the test generally only requires the use ofthe machines for a short period of time, say, a few days or weeks atmost. Accordingly, by performing the test in this virtual lab, theengineer avoids the need to requisition actual hardware and thecorresponding large expense, as would generally be required to test thesite in a non-virtual environment.

In addition to eliminating the need to requisition physical hardware,the virtual lab service also enables the engineer to quickly configureand launch a test environment. In contrast, tests using actual machinesoften require hours of setup time for connecting machines, installingsoftware, booting up the machines, and so on. In selected embodiments ofthe virtual lab service, virtual machines can be requisitioned andlaunched without requesting use of the virtual machines in advance. Thisis typically possible in a MTVMI where the amount of available computingpower is large in relation to the instantaneous user demand.

FIG. 10 shows a projects tab 1005 within the virtual lab service.Projects tab 1005 provides information 1010 regarding a particularproject (“Security Verification”) accessible to the engineer, virtualmachine configurations 1015 available within the project, runningconfigurations 1020 within the project, and software assets 1025associated with the project. Each of these features will be described inadditional detail below with reference to the remaining figures.

FIGS. 11 and 12 illustrate different virtual data centers available tothe engineer. As seen in the figures, these virtual data centers eachinclude a number of virtual machines. Each virtual machine is visuallyrepresented by a thumbnail image of the console display currently beinggenerated by the virtual machine. For illustration purposes, each of thevirtual machines has been configured with a unique combination ofcomputing characteristics, operating system, and status. For instance,in FIG. 11, a web server 1105 is implemented by a virtual machine havinga 1 GHz CPU, 1 GB of RAM, and running the Microsoft Windows operatingsystem. Within a virtual network, web server 1105 has a network address“192.168.0.1” and a status “running”. Other virtual machines havedifferent configurations, including different operating systems,different statuses, and so forth. Once machines are up and running inthe illustration of FIGS. 11 and 12, the engineer can click on athumbnail image of any one of the virtual machines to interact with themachine through the engineer's local interface.

In the examples of FIGS. 11 and 12, the status of each virtual machine(e.g., running, booting, resuming) can be indicated by both a textualindicator and a background color around the corresponding virtualmachine icon. The virtual lab service may allow users to modify virtualnetwork settings such as domains, subnet attributes and IP addresses ofvirtual machines within a particular lab.

In some embodiments, the virtual lab service enables the user togenerate a snapshot of one or more running virtual machines, such as asingle virtual machine, all of the virtual machines in a virtualnetwork, or all of the machines in a configuration. For example, theuser may select a visual control associated with the machine, network,or configuration in order to generate a snapshot. When a snapshot isgenerated, complete state for each of the virtual machines is stored inthe library. After the snapshot is generated, this item may be selectedfrom the library in order to instantiate any number of new instances ofthe set of virtual machines that were the subject of the snapshot. Thisis useful, for example, to establish a large number of initiallyidentical instances of a particular configuration. Also, in a testingproject, it can be useful for a tester who has just produced an error ina configuration being tested to establish a snapshot of theconfiguration that can be reviewed later by a developer to determine theconditions under which the error occurred.

In some embodiments, the virtual lab service allows the user to generatefor a particular virtual machine a URL that can be used, in the absenceof any other identifying information, to access the console display forthat virtual machine. This URL can be distributed to users that are notauthorized under the tenant's account to provide access to thisparticular virtual machine to those users. Such access may permit onlyobserving the console display of the virtual machine, or it may permituser interface interactions with the virtual machine. In someembodiments, the user can click on a CD icon displayed by the MTVMI inconnection with a virtual machine (not shown) in order to insert asoftware installation file such as an ISO file from the assets list intothe virtual machine allowing the user to install additional software onthe virtual machine.

FIG. 13 is a display diagram showing a sample display typicallypresented by the infrastructure in order to permit a user to control theoperation of virtual machines. The display 1300 shows information 1310and 1320 for two virtual machines. The information 1310 for the firstvirtual machine includes information describing the virtual machine, aswell as the following controls: a run/suspend control 1311 for beginningor suspending execution of the virtual machine; a shut down control 1312for invoking an operating system function on the virtual machinedesigned to turn the computer system running the operating system off; apower off control 1313 for immediately disconnecting power to thevirtual machine; a Remote Display Protocol (RDP) for accessing amachine's console display over the network from a remote location; aconnection control 1314 for establishing an RDP connection with thevirtual machine; an ISO control 1315 for loading a software installationISO file from the software assets list as a CD-ROM into the virtualmachine; a delete control 1316 for deleting the virtual machine; and asettings control 1317 for editing network and hardware attributes of thevirtual machine. Each of controls 1311-1317 operate specifically withregard to the virtual machine identified by virtual machine information1310. The display also includes controls 1391-1394 which, when operatedby the user, operate with respect to every virtual machine whose checkbox (e.g., check boxes 1319 and 1329) is checked. The display alsoincludes controls 1381 and 1382 for respectively selecting all of thevirtual machines and deselecting all of the virtual machines.

FIG. 14 is a display diagram showing a sample display typicallypresented by the infrastructure to display information about and allowinteraction with virtual data center configurations available to a userin their Lab. The display 1400 includes entries 1410, 1420, and 1430each corresponding to a different virtual data center configuration thatis accessible to the user. In a manner similar to that discussed abovein connection with FIG. 13, these entries contain check boxes 1411,1421, and 1431, respectively. Each of these can be checked so thatcontrols 1491, 1492, 1493, and 1496 operate with respect to thecorresponding virtual machine. The display also includes controls 1494and 1495 for switching between a list view of configurations and a tilesview of configurations, respectively.

FIG. 15 is a display diagram showing a sample display typicallypresented by the infrastructure to display information about and permitmanipulation of virtual data center configurations in a tiles view. Thedisplay 1500 includes sections 1510, 1520, and 1530, each containinginformation about a different virtual data center configuration. Forexample, information 1510 about the first virtual data centerconfiguration shows two virtual machines, and includes controls 1512,1513, and 1514 whose operation affects each of the virtual machines inthe first configuration. Information 1510 further includes a check box,like check boxes 1521 and 1531 for other machine configurations, thatdetermines whether controls 1591, 1592, 1593, and 1596 apply to theconfiguration.

FIG. 16 is a display diagram showing a sample display that is typicallypresented by the infrastructure in order to permit the user to specifyautomatic suspension of the virtual machines for a machineconfiguration. The display 1600 may be reached by selecting anauto-suspend control 1602. The display includes information 1610identifying the machine configuration. The infrastructure permitsselection between the following options for suspending this machineconfiguration: an option 1621 not to automatically suspend this machineconfiguration; an option 1622 for suspending the machine configurationafter all of the virtual machines in the configuration have been idlefor at least a user-specified amount of time; and an option 1623 tosuspend the virtual machines of the machine configuration at auser-specified time and date. The display also includes an updatecontrol 1630 for updating the auto suspension settings for this machineconfiguration.

FIG. 17 is a display diagram showing a sample display typicallypresented by the infrastructure in order to publish VM URLs foraccessing virtual machines of a machine configuration. The display 1700can be reached by selecting a Publish VM URLs control 1701. The displayenables the user to determine with option 1710 whether URLs aregenerated and activated for accessing the consoles of the virtualmachines of the current machine configuration. Where option 1710 isselected, the user can elect between option 1711 in which passwords arenot required in order to access a virtual machine using its URL and 1712where a user-specified password is required to access each virtualmachine in the machine configuration. Also, where the user selectsoption 1710, the user can determine whether to select option 1720 forallowing users who access a virtual machine in the machine configurationusing its published URL to control the virtual machine as contrastedwith merely viewing its console output. Also, where the user selectsoption 1710, the user may elect between option 1721 to allow those inpossession of a virtual machine's URL to access it at all times, andoption 1722 to allow access only during a user-specified time-of-dayrange. The display 1700 also includes a save control 1730 to updatethese attributes of the current machine configuration.

FIGS. 18 through 20 illustrate a library of virtual data centerconfigurations available to the engineer within the virtual lab service.The library includes configurations that have virtual machine imagespreinstalled with common operating systems, databases and applications.By creating a lab from pre-installed images, the user can save hours ofmanual installation tasks.

Configurations 1805 shown in FIG. 18 have all been installed in the labof FIG. 11. These and other configurations such as those illustrated inFIG. 19 can be installed in labs or added to currently runningconfigurations by actuating buttons within the interfaces shown in FIGS.18 and 19. Although the engineer may create labs from predefinedconfigurations, the engineer could alternatively assemble and configurenew virtual data center configurations from scratch by specifying a newcombination of virtual hardware, operating system, and so on. Theengineer could then add the new configuration to an existing lab orcreate a new lab through the virtual lab service. In general, any newconfigurations created by a user be associated with that user or sharedamong other users within the MTVMI.

Where a single user or tenant runs multiple virtual data centerinstances with identical virtual networks (e.g., multiple configurationswith same network subnet address, same IP addresses for virtualmachines) within the virtual lab service, the different instances aregenerally fenced off from each other. Fencing is typically accomplishedby using a virtual private network to establish a separate virtual LANfor each network instance. In some embodiments, the virtual machines ineach of the instances of the network are assigned the same set of IPaddresses. Accordingly, to provide independent access to the Internetfor the virtual machines of each of the instances of the network, thenetwork nodes and/or the firewall proxy requests from these virtualmachines to the Internet, enabling the proxy to route the reply from theInternet to the correct one of the virtual machines having the source IPaddress associated with the original request. In various embodiments,this proxying is performed in a variety of ways, such as mapping eachvirtual machine instance to a different public IP address, URL, orcombination or URL and port number.

Returning to FIG. 11, to test the e-commerce web site, the engineer runsthe web site on the virtual machine designated as web server 1105. Whilerunning the website, web server 1105 stores and retrieves data from avirtual machine designated as a database server 1115. Three virtualmachines having a variety of configurations are designated as testclients 1110 for generating traffic for the web-site. Finally, a virtualmachine designated as a test controller 1120 controls test clients 1110to perform the test.

FIG. 21 illustrates software assets 2105 (an asset can be, for example,any file containing text, data, software program, etc.) that have beenuploaded by the engineer to the virtual lab service. Using control 2110,the engineer can upload software assets from any machine (e.g., theirlaptop or other networked machine that they have access to includingvirtual machines running in the MTVMI) into an asset library in thevirtual lab service. Once the assets have been uploaded to a librarywithin the virtual lab service, the assets can be added to projects, asillustrated, e.g., by the information under the “assets” label in FIG.10. By adding the assets to a project, the assets become available toother users assigned to the project. This allows the engineer to sharethe assets with other users. Additionally, web browsers and other toolsexecuting on any machine (running within the MTVMI or elsewhere) may beused to “Download” software assets from the MTVMI asset library to themachine over the Internet. To perform this operation, the engineer canactuate the “Download” button from within their browser or another tool.

FIG. 22 shows an aspect of the virtual lab application that allows anadministrator to monitor use of the virtual lab service. To monitorspecific types of usage, the administrator may select various parametersfrom drop down menus in an area 2205. For instance, the administratormay select a certain type of activity to monitor, such as running avirtual machine, stopping a virtual machine, etc. Alternatively, theadministrator may choose to monitor activity of certain users, activityaround certain dates, and so on. Activity that falls within theparameters specified in area 2205 is displayed in an area 2210.

FIG. 23 is a display diagram showing a sample display typicallypresented by the infrastructure to show the consumption of resourcesrelative to quotas across an entire customer as well as by an individualuser. The display 2300 includes a section 2310 about consumptionrelative to quotas for the customer, and section 2320 for consumptionversus quotas for the current user. The rows in each section are dividedinto two columns: usage column 2301 that shows the amount that has beenused by the entity and limit column 2302 that shows the quota to whichconsumption is limited for the entity. The customer section 2310includes row 2311 for number of hours that virtual machines execute; row2312 for the number of virtual machines allocated; row 2313 for thevolume of persistent storage allocated; and row 2314 for the number ofpublic IP addresses allocated. Section 2320 for the individual userincludes corresponding rows 2321-2323, showing the user's consumptionand the user's limit as contrasted with the customer's total usage andlimit. The display also includes a more concise version 2350 of thisinformation that is suitable to be displayed on a variety of pagescontaining a variety of other information.

FIG. 24 is a display diagram showing a display typically presented bythe infrastructure to report on usage of resources including paidresources made accessible via the brokering service, described infurther detail below with reference to FIGS. 7 and 47-49. In a list ofresources 2410, 2460, and 2470, a paid item icon 2411 indicates usage ofa resource 2410 that incurs additional billable charges for the user.The entry includes information about the billable item includinginformation 2412 and 2416 identifying the item, information 2413identifying the vendor, information 2414 indicating the basis for whichthe item is metered for payment, and information 2415 indicating thecost of consuming the item. Usage entries 2460 and 2470 in the listreport usage of resources that are accounted against the customer anduser's resource consumption quota. Each entry includes information aboutthe time of resource consumption, the user that incurred the resourceconsumption, the amount of various resources consumed and the items(e.g., virtual data center configuration) whose usage resulted in theresource consumption.

In some embodiments, a tenant may specify specific levels of service tobe provided by the MTVMI. For example, the tenant may specify that itslabs run in a dedicated virtual machine host node, and/or access adedicated storage node; specify that physical processor utilization beno greater than a level such as 75% on the virtual machine host nodes onwhich its labs execute; or specify a maximum level of resources of oneor more types that the tenant can consume in a particular period oftime. In some embodiments, where quotas are specified for a tenant,either by the tenant or the operator of the MTVMI, the MTVMI displaysusage relative to the quota to one or more of the users for the tenant.

FIG. 25 shows an additional conceptual diagram illustrating theorganization of the MTVMI in some embodiments. FIG. 25 shows thephysical arrangement of hardware components of the MTVMI in someembodiments. The MTVMI infrastructure has one or more storage servers orstorage nodes, one or more server blades that serve as virtual machinehosting nodes or as network nodes all of which are housed in racks andconnected in an Ethernet network fabric. Additional servers on thenetwork fabric execute components of the MTVMI management application(e.g., shown as ETL Web FE in the figure). The MTVMI infrastructure inaddition includes other typical components such as router, VPN gateway,firewall, load balancer, etc.

FIG. 26 illustrates some of the network services provided in connectionwith virtual networks in some embodiments. These network servicesinclude, for example, DNS, DHCP, Firewall, NAT (Network AddressTranslation), VPN, Traffic shaping, networked storage etc. Theseservices are accessible to the virtual machines that are part of thisvirtual network. In some embodiments, network services in any virtualnetwork are fully isolated from network services in other virtualnetworks within the MTVMI.

FIGS. 27-46 are display diagrams showing displays presented by the MTVMIin some embodiments. FIG. 27 shows a display enumerating availablevirtual data center configuration templates that is presented in someembodiments. FIG. 28 shows a display presenting information about asingle configuration template that is presented in some embodiments.FIG. 29 shows a display for monitoring resource utilization within thevirtual machines in a virtual data center that is presented in someembodiments. FIG. 30 shows a display that a tenant can use to configuretheir user's access permissions that is presented in some embodiments.FIG. 31 shows a display that enables a tenant quota to be specified thatis presented in some embodiments. FIG. 32 shows a display for auditing aMTVMI user's operations across tenants that is presented in someembodiments. FIG. 33 shows a display for monitoring tenant resourceusage with respect to quotas that is presented in some embodiments. FIG.34 shows a display for monitoring the status of virtual machine hostnodes that is presented in some embodiments. FIG. 35 shows a display fora user to control network settings for a virtual machine that ispresented in some embodiments. This display shows fields for the user tochange the virtual machine's Hostname, IP address, MAC address, and alsoopen specific network ports in the firewall to allow access from thepublic Internet. FIG. 36 shows a display for reviewing and modifyingproject settings that is presented in some embodiments. FIG. 37 shows adisplay for navigating information associated with a virtual lab servicethat is presented in some embodiments. FIG. 38 shows a displayidentifying users associated with a tenant that is presented in someembodiments. FIG. 38 also shows a control that the tenant can actuate tocreate accounts for new users within the tenant's account. FIGS. 39 and40 are displays showing lists of virtual data center configurations thatare made available for users by a virtual lab service that is presentedin some embodiments. FIG. 41 shows a display identifying tenants that ispresented in some embodiments. FIG. 42 is a display identifying usersassociated with each tenant that is presented in some embodiments. FIG.43 is a display showing assets available within the virtual lab servicethat is presented in some embodiments. FIG. 44 is a display showinginformation about a virtual machine configuration that is presented insome embodiments. FIGS. 45 and 46 show displays of a virtual machine'sconsole display illustrating a user authenticating (FIG. 45) andactivating (FIG. 46) a third-party software license for softwareinstalled on the virtual machine. FIG. 46 shows a display for activatinga pre-installed instance of an operating system using a license keyowned by a tenant that is presented in some embodiments. It can be seenthat the display includes tabs, such as tabs “Windows Server 2003Standard” and “Windows Server 2003 Enterprise R2 SP2,” that each referto a different virtual machine. The user can select a view of aparticular machine by selecting the tab corresponding to that virtualmachine.

In some embodiments, MTVMI 100 provides a brokering service that enablestenants and users to exchange or offer products and/or services. FIG. 7is a flowchart illustrating the provision of a brokering service byMTVMI 100 in some embodiments. The brokering service of FIG. 7 isprovided by a brokering portal where tenants and users are allowed toexchange software and other data that may be useful within MTVMI 100.The data may include, e.g., custom virtual machine or virtual datacenter configurations, software applications, custom scripts or programswritten for MTVMI 100, to name but a few.

In step 705, MTVMI 100 provides a brokering portal. In one example, thebrokering portal comprises a software application that presents agraphical user interface within a user's display. The graphicalinterface may include a mechanism for presenting available softwareand/or data that the user can download. The graphical user interface mayfurther include a mechanism for allowing the user to upload software tothe portal. In some embodiments, the MTVMI grants access to the tenantto software that is already available within the MTVMI. The portal mayrequire the user to provide a form of payment to download certainsoftware or data. Alternatively, the software or data may be providedfree of charge.

In step 710, MTVMI 100 receives data from the user through the portaland stores the data in a common access area. For explanation purposes, auser who uploads data to the portal is referred to as a vendor in FIG.7, even though the portal does not necessarily require users to exchangepayment for sharing data. Similarly, a user who downloads software fromthe portal will be referred to as a buyer.

In step 715, MTVMI 100 presents data to potential buyers within theportal. In step 720, MTVMI 100 facilitates a transaction between thevendor and the buyer by, for example, allowing the buyer to downloaddata provided by the vendor. MTVMI 100 facilitates transactionsinvolving the exchange of money by, for example, enabling users to debitor credit bank accounts, providing deposit accounts within the MTVMI,adding incurred expenses to a tenant's MTVMI account, etc.

FIGS. 47-49 are display diagrams illustrating a user interface designedfor use by vendors offering items for sale via the brokering portal insome embodiments. FIG. 47 is a display diagram showing a sample displaytypically presented by the infrastructure in order to register a newvendor for the brokering portal. The display enables a user associatedwith the vendor to enter the following information about the vendor: thecompany name 4701 of the vendor; the company address 4702 of the vendor;the contact name 4703 of a contact at the vendor; the email address 4704of the contact; a phone number 4705 of the contact; and a password 4706to be used by the vendor to access a vendor portal for interacting withthe brokering portal. The user can select a submit control 4710 togenerate a profile that is based upon the inputted information.

FIG. 48 is a display diagram showing a sample display typicallypresented by the infrastructure in order to enable a vendor to list anew item for sale in the brokering portal. The display 4800 is typicallymade available to a user associated with the vendor in response to thatuser signing into the vendor portal on behalf of the vendor. The displayincludes information 4801 identifying the vendor. The user can use thedisplay to input the following information about the item to be listed:the name 4811 of the item; a description 4812 of the item; a supportlink 4813 for the item; and a date 4814 on which the listing willexpire. The user may also input pricing for any of a number of differentpricing schemes, including the per month cost 4821 for each user whouses the item; the per year cost 4822 for each user who uses the item;the per hour charge 4823 for the item; and the one-time charge 4824 peruser for using the item. Typically, if the user does not enter a pricein one of these fields, the option is not offered as a basis on whichbuyers can purchase the item. The display further includes controls 4831and 4832 for specifying a path to the file containing the item that isto be uploaded to the infrastructure from the vendor's computer system.The user can select submit control 4840 in order to create a listing inaccordance with the inputted information.

FIG. 49 is a display diagram showing a sample display typicallypresented by the user in order to account to a vendor for revenuegenerated by the vendor's items in the brokering portal. The display4900 shows a period 4901 such as a month, to which the displayedaccounting applies. The user can select controls 4902 or 4903 tonavigate to display an accounting for other periods of time. The displayincludes a table in which each row corresponds to a particular basis forcharging customers for a particular item. These rows are grouped intogroup 4920 for the Crossbow Debugger item and group 4930 for the ImageCropper item. Each row is divided into the following columns: an itemcolumn 4911 identifying the item; a rate column 4912 identifying therate charged to users for the payment option to which the rowcorresponds; a units column 4913 showing the number of units of thatcharging option that were consumed by customers during the selectedperiod; a revenue column 4914 showing the revenue produced in connectionwith the current item and charging option during the selected period;and an item title total column 4915 showing, for the last row of eachgroup, the amount of revenue generated by the item during the selectedperiod. The display also includes an indication 4940 of the grossrevenue for the selected period, and an indication 4950 showing theportion of the revenue that is retained to support the operation of thebrokering portal, and an indication 4960 of the net revenue to thevendor for the selected period.

FIGS. 50-53 describe the architecture of some embodiments of the MTVMImanagement application (shown in FIG. 6) to realize its functionalcapabilities.

FIG. 50 is a block diagram illustrating a data model representation ofvirtual data centers in an MTVMI application in some embodiments. Eachvirtual data center is represented at the highest level by a virtualdata center configuration 5001. Virtual data center configuration 5001has an ID that is a globally unique reference to the specific virtualdata center configuration. Each virtual data center configuration isassociated with one or more virtual machine objects 5021, one or morevirtual networks 5011, an autosuspend attribute 5002 and a VM URL (orpublish URL attribute) 5003. The VM URL attribute specifies a URL handlethat can be used to reference each of the virtual machines in thevirtual data center as well as associated attributes such asauthentication password, user privileges on operating the virtualmachine. The autosuspend attribute specifies the policy that the userhas configured for suspending the virtual data center or its componentvirtual machines (e.g., policies include after a specific idle duration,at a specific wall clock time, etc.) Each virtual network object 5011has several attributes including Subnet specification 5012, DNSspecification 5013, DHCP specification 5014, VPN specification 5015,Network Drive specification 5016, and Traffic specification 5017. TheSubnet attribute 5012 can specify the network address and network subnetmask associated with that network. DNS and DHCP attributes 5013 and 5014can specify the network address of these services and whether theyshould be the default services or custom services provided by the user.VPN attribute 5015 can specify the configuration attributes of thenetwork tunnel including the identity of the remote VPN gateway, networkaddress specifications for the tunnel, access credentials, policy forauto-connecting the tunnel etc. Network Drive specification 5016 canspecify a reference to one or more virtual storage drives that should beaccessible in the network. Traffic attribute 5017 can specify abandwidth quota for the network. Each virtual machine object 5021 has anID that globally uniquely identifies that virtual machine. Each virtualmachine object is associated with a Display attribute 5022, a CPU/RAMattribute 5023, one or more Network Interface objects 5031, and one ormore Disk objects 5041. The Display attribute provides a reference tothe graphical console display of the particular virtual machine. TheCPU/RAM attribute specifies the number of processing elements and theamount of RAM for the particular virtual machine. Each Network Interfaceobject 5031 is bound to one of the Network Objects associated with theparent virtual data center configuration. It also specifies severalattributes including its MAC address, one or more IP addresses for theinterface (e.g., Public IP 5033 routable from the public Internet,Private IP 5034 routable only within the Network that the interface isbound to, and also one or more firewall rules identifying which networkports (e.g., port 22 for SSH, port 80 for HTTP etc.) are open for accessfrom the public Internet. Each Disk object 5041 has a globally unique IDthat identifies the disk file or files that hold the content of thisdisk on the storage nodes.

The virtual data center data model discussed in FIG. 50 is an exampleembodiment. Other embodiments include additional attributes, bothattributes that are specified by the user (e.g., name for the virtualdata center, specific tasks to be executed at specified times on thevirtual data center) and attributes that are determined by the MTVMImanagement application itself (e.g., other IDs or version numbers). Inaddition, the MTVMI management application will maintain runtimeinformation about the virtual data center, including its executionstate, error states if any for any of the virtual data centercomponents, locking state to support serialization of operations on thevirtual data center, queue of pending operations on the virtual datacenter, and timestamps for last user activity on the virtual datacenter.

A user of the MTVMI can provide various inputs, as discussed overprevious embodiments, to specify the desired configuration for a virtualdata center including its virtual machines, its virtual network, virtualstorage, etc. The MTVMI management application instantiates a virtualdata center configuration object with these configuration parameterswhen creating the virtual data center for the user. When performingsubsequent user-requested operations on the virtual data center, theMTVMI management application will perform these operations across thebreadth of the attributes and state of the virtual data center asappropriate for the specific operation. For example, one of theoperations that the MTVMI management application supports is to Snapshotthe virtual data center. This Snapshot operation captures the completeattributes and state of the virtual data center. This enables the userto subsequently perform a second operation to create a duplicate copy ofthe virtual data center from the snapshot. The duplicate copy that iscreated is identical to the original but is however independent andisolated from the original.

The MTVMI management application orchestrates several steps, some donein sequence and some done in parallel, to perform operations on thevirtual data center. To perform a Snapshot operation on the virtual datacenter, the steps performed include quiescing previous user-initiatedoperations on the virtual data center, quiescing execution of thevirtual machines in the virtual data center so that a consistentcheckpoint of the virtual machine states can be saved, saving the stateof the executing virtual machine including the state of its processorsand memory, saving a copy of disk data that is associated with thevirtual machines and any network share drives, creating a new virtualdata center configuration object with a newly assigned globally uniqueID, and saving all information that needs to be persisted from theoriginal virtual data center configuration object to the newly createdvirtual data center configuration object. Virtual data centerconfiguration object information that is saved include Virtual Machineobjects with newly assigned globally unique ID, the CPU/RAM attributesof each virtual machine, the Network Interface attributes of eachvirtual machine including their Firewall, Public IP, and Private IPattributes, Disk object information referencing the IDs of the newlysaved copies of the disk state, Network object information including itsattributes discussed in FIG. 50, the Autosuspend attribute and the VMURL attribute. Specific attributes that need to be uniquely associatedwith the new virtual data center configuration object are not saved,including globally unique IDs associated with the original virtual datacenter configuration object, its virtual machines, and its Disk Objects,VM URL references to the virtual machines in the original virtual datacenter configuration and Display object references to the virtualmachines in the original virtual data center configuration.

When a user initiates an operation to instantiate a Snapshot copy of avirtual data center, in some embodiments, the MTVMI managementapplication orchestrates the necessary steps to instantiate the virtualdata center. This includes instructing the Network Manager component toassign a dedicated VLAN for each of the Network objects referenced inthe virtual data center configuration and create dedicated instances ofnetwork services in each of the VLANs with configuration parameters asspecified with the attributes of the Network object. This also includesinstructing the Storage Manager to make the required disk objectsaccessible to the virtual machines that will be instantiated in thevirtual data center. This also includes instructing the VM Manager toallocate resources for running the virtual machines, configuring theCPU/RAM and network interface attributes of the virtual machines to meetthe specifications in the virtual data center configuration object,triggering the execution of the virtual machines from their associatedprocessor, memory, and disk state. This also includes establishing theAutosuspend, VM URL, and Display controls and references to meet thespecifications in the virtual data center configuration object. Theseand other steps orchestrated by the MTVMI management applicationinstantiate an exact duplicate copy of the original virtual data centerwhich is, however, independent and isolated from the original virtualdata center.

In other embodiments, the MTVMI management application supports andorchestrates additional operations including creating a copy from theSnapshot without instantiating a virtual data center, creating aduplicate instance of a virtual data center directly from aninstantiated virtual data center without creating an explicit Snapshot,etc.

The MTVMI management application is comprised of a number of componentprocesses as discussed previously. FIG. 51 is a block diagramillustrating component processes in the MTVMI management application insome embodiments. The MTVMI management application includes one or moreinstances of Webserver process 5101, Display Proxy process 5102,Autosuspend process 5103, Scheduled Task process 5104, Quota Inspectorprocess 5105, Database process with one or more associated databases5105, Storage Manager process 5106, Network Manager process 5107, and VMManager process 5108. The Webserver process 5101 services user requestsover a standard protocol such as HTTP or HTTPS that is carried over theInternet. Servicing these requests cascade to requests for otherprocesses in the MTVMI management application as appropriate dependingon the request. The Display Proxy process 5102 enables the user toaccess the graphical console display of virtual machines in theirvirtual data centers running in the MTVMI. When a user requests accessto the graphical console display of the virtual machine, the Web serverprocess first retrieves the Display object reference for the associatedvirtual machine from the virtual data center configuration object andreturns it to the user's browser. An applet is triggered to run on theuser's browser which supplies this Display object reference to the Webserver in a second request. This request is passed to the Display proxyprocess which associates the supplied Display object reference with thegraphical console port of the specific virtual machine and bridges theconnection between the user's browser applet to the graphical console ofthe virtual machine. The graphical console is rendered on the user'sbrowser over the Internet through the use of a standard remote displayprotocol carried over the Internet such as VNC (Virtual Network Console)or RDP (Remote Display Protocol). The Autosuspend process 5103 runsperiodically to examine expiry of idle time intervals or other timingintervals for virtual data centers and if these intervals have expiredtriggers appropriate additional processes for further action on thevirtual data center (e.g., to suspend execution of all virtual machinesin the virtual data center). The Scheduled Task process 5104 triggersthe execution of additional processes at pre-specified time of day toperform specific actions on the virtual data center (e.g., to initiateexecution of all virtual machines in the virtual data center at thespecified time). The Quota Inspector 5105 process runs periodically todetect exhaustion of time-dependent resource usage quotas such as thelength of time a virtual machine is allowed to run. The Quota Inspectorprocess examines events with ongoing resource usage (to be discussedfurther with reference to FIG. 53) and the entities to which theresource usage is assigned to (e.g., entities such as customer, user,project). If the aggregate of prior resource usage and the ongoingresource usage is determined to exceed the resource quota allocated forthe entity, the Quota Inspector triggers the execution of additionalprocesses to perform further action on the virtual data center. Theseactions can include suspending the execution of virtual machinesassociated with that entity to stop further usage, sending notificationto one or more users associated with the entity informing them ofexceeding quota, posting error notifications in the virtual data centerconfiguration object, automatically increasing the resource quotaallocation etc. The Database process 5106, as the name indicates,maintains metadata information used by the MTVMI management applicationin a database and services queries and operations on this metadata. Thismetadata includes information about entities such as Customer, User,Project, virtual data center configurations, Library, etc. The StorageManager process 5107, Network Manager process 5108, and the VM Managerprocess 5109, which have been discussed previously, manage the creation,operation, and management of the virtual storage, virtual network, andvirtual machines provided for each of the virtual data centers managedby the MTVMI.

In some embodiments, MTVMI management application 115 supports accesscontrol mechanisms to enable tenants to restrict their users' accessrights to specific objects (e.g., virtual data centers, virtualmachines, data files, etc.) within the MTVMI as described earlier. FIG.52 is a block diagram illustrating the organization of the data modelthat supports access mechanisms of MTVMI management application in someembodiments. This data model includes a Customer Table 5201, a UserTable 5202, a Project Table 5203, a Role Assignment Table 5204, and aShares Table 5205. The Customer Table 5201 records a globally uniqueCustomer ID for each tenant that has a customer account on the MTVMIsystem and associated attributes. These customer attributes include, forexample, the customer's name, their address, contact information for thecustomer's primary contact, the quota allocation for the customer foreach consumable MTVMI resource (e.g., virtual machine hours, storagesize, network bandwidth, etc.), a globally unique quota ID for thecustomer, etc. The User Table 5202 records a globally unique ID for eachuser and associated attributes. These user attributes include, forexample, the user's name and address, the customer ID of the customeraccount (i.e., tenant) that the user is associated with, the user'slevel of privilege within the customer account (e.g., accountadministrator), the user's credentials for accessing the MTVMI, thequota allocation for the user for each consumable MTVMI resource, aglobally unique quota ID for the user, etc. The Project Table 5203records a globally unique ID for each Project in the MTVMI andassociated attributes. These Project attributes include, for example,the name of the Project, the customer ID of the customer account towhich the Project belongs, the user ID of the user who created theProject, the quota allocation for the Project for each consumable MTVMIresource, a globally unique quota ID for the Project etc. The RoleAssignment Table 5204 records the privileges of users for each Projectto which they have access rights. For every Project that user has accessto, Role Assignment Table 5204 tracks the user's ID and the Project's IDand the user's role within the Project. The MTVMI management applicationrecognizes multiple tiers of roles each of which awards different rightson different objects in the MTVMI. For example, a Member tier role in aProject allows the user to read and use objects (e.g., virtual datacenter) that are already assigned to the Project but does not allow theuser to add or remove objects from the Project. The Shares Table 5205records for each object in the MTVMI, the ID of the object (e.g.,virtual data center configuration ID for a virtual data center), the IDof the user who owns the object, the ID of the customer account to whichthe object belongs, and the list of IDs of Projects to which the objectis assigned (objects can be assigned to multiple Projects in someembodiments).

In some embodiments, MTVMI management application 115 supports accesscontrol to MTVMI objects using the data model described in FIG. 52. Whena user attempts to perform an operation on a specific object (e.g., totake a Snapshot of a virtual data center), the MTVMI managementapplication permits the operation to proceed successfully if it meets atleast one of the following conditions: 1) the user ID of the usermatches the owner ID for the object in the Shares Table 5205, or 2) theuser has administrator level privileges in the customer account thatcontains the object, which is verified using the customer ID associatedwith the object in the Shares Table 5205 and the customer ID and theuser's privilege level for the user's ID in the User Table 5202 or 3)the user has the requisite access rights within a Project to which theobject is assigned, which is verified using the user's role indicated inthe Role Assignment Table 5204 for each Project ID that is included inthe list of Project IDs associated with the object in the Shares Table5205.

In some embodiments, MTVMI management application 115 supports theallocation and enforcement of usage quota mechanisms for variousconsumable MTVMI resources (e.g., virtual machine hours, storage bytesused, network bandwidth used etc.). The MTVMI application supports thisquota management for multiple types of entities including customeraccounts (customers for simplicity), individual users, and individualProjects. FIG. 53 is a block diagram illustrating the organization ofthe data model that supports usage quota mechanisms of MTVMI managementapplication in some embodiments. This data model includes a CustomerTable 5301, a User Table 5302, a Project Table 5303, and a ResourceUsage Events Table 5304. The Customer Table, the User Table, and theProject Table are identical to their previous descriptions withreference to FIG. 52. The Customer Table 5301 records a globally uniqueCustomer ID for each tenant that has a customer account on the MTVMIsystem and associated attributes including the quota allocation for thecustomer for each consumable MTVMI resource (e.g., virtual machinehours, storage size, network bandwidth, etc.) and a globally uniquequota ID for the customer. The User Table 5302 records a globally uniqueID for the user and associated attributes including the quota allocationfor the user for each consumable MTVMI resource and a globally uniquequota ID for the user. The Project Table 5303 records a globally uniqueID for each Project in the MTVMI and associated attribute including thequota allocation for the Project for each consumable MTVMI resource anda globally unique quota ID for the Project, etc. The Resource UsageEvents Table 5304 records an entry for each Event in the MTVMI thatinitiates or terminates usage of any consumable resource in the MTVMIsystem. For example, when a user instantiates a virtual data center, theinitiation of each virtual machine within that virtual data center is anEvent that initiates the consumption of virtual machine processor andmemory resources. Another example is when a user deletes a data filefrom their account within the MTVMI system, this is an Event thatterminates the consumption of storage resource within the MTVMI. Foreach such Event, the Resource Usage Events Table 5304 maintains aglobally unique ID for the Event, the Type of resource that is consumed(e.g., virtual machine processor resource, storage capacity resourceetc.), the size of resource consumption (e.g., the number of virtualmachine processors, the amount of storage in bytes etc.), the start timeof the Event which is populated when the resource usage is initiated,the end time of the Event which is populated when the resource usage isterminated, the ID of the object that is associated with the resourceconsumption (e.g., virtual machine ID, the Disk ID, the data file IDetc.), and the quota IDs of all entities that consumed the resource(i.e., user quota ID of the user who triggered the resource usage event,the customer ID of the customer account the user belongs to, the ProjectID of any Projects to which the object is assigned).

In some embodiments, MTVMI management application 115 supportsenforcement and reporting of usage quotas for various consumable MTVMIresources using the data model described in FIG. 53. The contents of theResource Usage Events Table 5304 contains the total history of resourceusage by any user, customer, or Project. The MTVMI managementapplication queries this database of information to generate variousreports or views, such as the current active usage of specific resourcetypes (e.g. virtual machine processors) by specific entities (e.g.,user, customer, Project), the aggregate usage of specific resource typesby specific entities within a specific time interval etc. In someembodiments, these queries can be triggered by user input (to generatespecific usage reports for the user), by MTVMI administrators togenerate billing invoices to tenants, or by the MTVMI managementapplication itself to automatically generate reports for users or toenforce resource usage quota limits. In some embodiments, the MTVMImanagement application periodically runs the necessary queries tomaintain and provide to users an update tally of resource usage by eachentity (e.g., user, customer, Project) for each type of resource. Insome embodiments, the MTVMI management application can enforce quotas onresource usage using two mechanisms. With the first mechanism, when auser's operation initiates resource usage, the sum of the updatedresource usage tally for the user, customer, and Project associated withthe operation and the incremental resource usage required by theoperation are compared against their respective resource usage quotas.The operation is allowed to continue if the resource quota is greaterthan the sum of the resource usage tally and the incremental resourcerequired by the operation. Otherwise, the operation is aborted withappropriate notification to the user. With the second mechanism, a QuotaInspector process (discussed previously with reference to FIG. 51) runsperiodically to perform quota enforcement. The Quota Inspectorcalculates the most current resource usage tally for each user,customer, and Project in the MTVMI and determines if any of these usagesexceed their respective resource quota. If a quota is exceeded for anyentity, the Quota Inspector queries the Resource Usage Events Table 5304for any Events belonging to that entity that will continue to increaseresource usage with the passage of time (e.g., virtual machine executiontime). For each such Event, the Quota Inspector triggers the executionof another process to perform an operation on the associated object toterminate the continued usage of resources (e.g., an operation tosuspend a running virtual machine). These embodiments illustrate how theMTVMI management application supports resource usage quotas, theirreporting, and their enforcement.

FIG. 54 is a flowchart illustrating the operations performed by acomponent of the MTVMI management application to instantiate a virtualdata center configuration on the MTVMI infrastructure in someembodiments. These operations perform steps to instantiate virtualmachines, virtual networks, and virtual storage for the virtual datacenter if the user has the required access rights and available resourcequota. In some embodiments these steps are performed through theindividual operations of various processes in the MTVMI managementapplication. In block 5405, the component records user input relating tothe Virtual Data Center Configuration in a database. In block 5410, thecomponent verifies that the user has the required access rights andavailable resource quota. In block 5415, the component instantiates diskimages of virtual machines storage nodes. In block 5420, the componentallocates resources on physical machines for the virtual machines. Inblock 5425, the component allocates dedicated VLANs for the Virtual DataCenter. In block 5430, the component configures network attributes ofvirtual machines in configuration files and in hypervisors on thephysical machine. In block 5435, the component configures network rulesfor the virtual machines. In block 5440, the component instantiatesnetwork services for each VLAN. In block 5445, the component enableshypervisor access to virtual machine disk images. In block 5450, thecomponent commands the hypervisors to run the virtual machines. In block5455, the component records references to console ports of the virtualmachines on the hypervisors. In block 5460, the component records anindication of resources consumed by the Virtual Data Center.

FIG. 55 is a flowchart illustrating the operations performed by acomponent of the MTVMI management application to save the state of theVirtual Data Center in some embodiments. In block 5505, the componentreceives from a user an indication to save state of the Virtual DataCenter. In block 5510, the component verifies that the user has therequired access rights and available resource quota. In block 5515, thecomponent commands hypervisors to suspend the virtual machines. In block5520, the component shuts down network services deployed for the VirtualData Center. In block 5525, the component unconfigures rules on networknodes related to the Virtual Data Center. In block 5530, the componentdeallocates VLANs allocated for the Virtual Data Center. In block 5535,the component creates snapshots of virtual machine state and disk imageson storage nodes. In block 5540, the component copies Virtual DataCenter configuration information into a new record in a database. Inblock 5545, the component updates a database record with references tovirtual machine state and disk images on the storage nodes.

In view of the foregoing, it should be appreciated that, while selectedembodiments have been described herein for illustration purposes,various modifications may be made without deviating from the scope ofthe invention. Accordingly, the invention is not limited except as bythe appended claims and the elements explicitly recited therein.

We claim:
 1. A method in a computing system including at least oneprocessor, comprising: for each of a plurality of virtual machines,displaying a visual indication of the virtual machine that includes aset of one or more graphical controls configured to manage the virtualmachine; and displaying a set of general graphical controls configuredto synchronously manage two or more of the virtual machines of theplurality, such that a single activation of one of the displayed generalgraphical controls results in a control action being performed withrespect to each of the two or more virtual machines of the plurality. 2.The method of claim 1 wherein the displayed set of graphical generalcontrols and displayed set of individual graphical controls all includea control for beginning execution in one or more virtual machines. 3.The method of claim 1 wherein the displayed set of graphical generalcontrols and displayed set of individual graphical controls all includea control for suspending execution in one or more virtual machines. 4.The method of claim 1 wherein the displayed set of graphical generalcontrols and displayed set of individual graphical controls all includea control for ending execution in one or more virtual machines.
 5. Themethod of claim 1 wherein the displayed set of graphical generalcontrols and displayed set of individual graphical controls all includea control for persistently storing the state of one or more virtualmachines.
 6. The method of claim 1 wherein the displayed set ofgraphical general controls and displayed set of individual graphicalcontrols all include a control for deallocating one or more virtualmachines.
 7. The method of claim 1 wherein the displayed set ofgraphical general controls and displayed set of individual graphicalcontrols all include a control for changing settings for one or morevirtual machines.
 8. The method of claim 7 further comprising,simultaneously with displaying the visual indications of the virtualmachines, displaying (1) a first graphical control for changing theselection controls of all of the virtual machines of the plurality to aselected state and (2) a second graphical control for changing theselection controls of all of the virtual machines of the plurality to adeselected state.
 9. A computer-readable storage device having contentsadapted to cause a programmed computer system comprising a processor, adata storage system, at least one input device, and at least one outputsystem, to perform operations, the operations comprising: for each of aplurality of virtual machines, displaying a visual indication of thevirtual machine that includes a set of one or more graphical controlsconfigured to manage the virtual machine; and displaying a set ofgeneral graphical controls configured to synchronously manage two ormore of the virtual machines of the plurality, such that a singleactivation of one of the displayed general graphical controls results ina control action being performed with respect to each of the two or morevirtual machines of the plurality.
 10. A computer-readable storagedevice storing contents adapted to cause a computing system including atleast one processor to perform a method for managing virtual machines,the method comprising: for each of a plurality of virtual machines,displaying a visual indication of the virtual machine that includes aset of one or more graphical controls configured to manage the virtualmachine; and displaying a set of general graphical controls configuredto synchronously manage two or more of the virtual machines of theplurality, such that a single activation of one of the displayed generalgraphical controls results in a control action being performed withrespect to each of the two or more virtual machines of the plurality.11. A virtual computing device comprising: a processor; and a storagedevice having instructions executable by the processor to: for each of aplurality of virtual machines, displaying a visual indication of thevirtual machine that includes a set of one or more graphical controlsconfigured to manage the virtual machine, and displaying a set ofgeneral graphical controls configured to synchronously manage two ormore of the virtual machines of the plurality, such that a singleactivation of one of the displayed general graphical controls results ina control action being performed with respect to each of the two or morevirtual machines of the plurality.
 12. The virtual computing device ofclaim 11 wherein the displayed set of general graphical controls anddisplayed set of individual graphical controls all include a control forbeginning execution in one or more virtual machines.
 13. The virtualcomputing device of claim 11 wherein the displayed set of generalgraphical controls and displayed set of individual graphical controlsall include a control for suspending execution in one or more virtualmachines.
 14. The virtual computing device of claim 11 wherein thedisplayed set of general graphical controls and displayed set ofindividual graphical controls all include a control for ending executionin one or more virtual machines.
 15. A virtual computing devicecomprising: a means for displaying a visual indication for each of aplurality of virtual machines, wherein the display includes a set of oneor more graphical controls configured to manage one or more of thevirtual machine; and a means for displaying a set of general graphicalcontrols configured to synchronously manage two or more of the virtualmachines of the plurality, such that a single activation of one of thedisplayed general graphical controls results in a control action beingperformed with respect to each of the two or more virtual machines ofthe plurality.